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A procedure to r<^late funding decisions and project 
evaluations in an annual grant progran fci vocational education 
administered by a state education department (SED) is described. 
Objectives of the study included developing a set of predictor 
variables for funding decisions^ and cutccme variables for evaluating 
decisions; determining consensus among funding decision makers are~~' 
applicants so one estimate of the validity of the major variables; 
and testing the feasibility of obtaining necessary data to implement 
a procedure linking funding and evaluation data, A first estimate of 
validity was obtained by examining the relationship between the SED 
and local education agency directors^ ratings and rankings of outcome 
and predictive impact variables. Feasibility of obtaining 
distribution data was also examined. Combining distribution data and 
judgmental data was the recommended prccedure for funding and 
evaluation purposes. The procedure is based on a series of 
objectively and subjectively known data which are revised 
systematically and which provide standards acceptable to both 
decision makers and fund applicants. (Author/MH) 
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Procedures are needed to relate evaluation findings and funding decision making. 
This paper describes a procedure that has been developed for a state education 
department's (SED) funding of vocational education programs. A first estimate 
of validity has been obtained by examining the relationship between the SED and 
local education agency directors' ratings and rankings of outcome and "predictive" 
impact variables. Feasibility of obtaining distribution data was also examined. 
The recommended procedure is to combine distribution data (as in the'TJresent 
study) and judgmental data, as used by Ory, Harris, Dueitt, and Clark (1978), 
for funding and evaluation purposes. 
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Objectives 

The geoal of ."ite present study was to defies a * ledure co" ri2late fuii:2i:^ Tiecis±crs 
and project: evalisatc-ons in an annual grant pras^a:; T^catlan^ educatloi:: adnrrnis t eipg? 

at the stance le^;^!- The first objective of ti£^ sti£n7r to ccrvr^wdjop a set bath 
"predictive"' -rari afe r i^s (those that can be knosOT- a£ "i.-me of rrirading) s^mrf mtcnme 
variables, rar ^-i»tHtig and ev^uation decisic 3 ^rprgycrrxBEily , A s^ond obj.iectlve wa:s 
to determine lijfe cmifsensus anwrng funding deci.-ir> ..^iateE^- 2nd f*in«£±ng appiicsjits s 
oae estimate '* f^^*^ ^lidit the major var^^iiO^ .to : 3 usea it: ^indir^r decision 
making and evr^sUoj.. . .-tr; ^.le tr rd major objec-\vr^ ' -3 test th-- ^easibl Ilrry of 
obtaln±ng dat^ ne ded ^- implr at a procedure: t ! < :mdi:sg Gi£r"±slon5 acd 
evaluation dsna. 

The procedms xc nk iv^j^^^ifl/tirm and fundiTV^ sions raquirres der-esEining rhe 
priority and cr±^'jv ighte i^ajor predicr^^^' d outcome rnrpact v^&nables, and 

prsnidlng estiinatca. a&ca^^i?0/^^ on which ,m might be descri;i»ed^or each of 

th& impact scales vgcrt:^ ^r^ve c 2d outcome). liair *^ :>r a variaole dfiEte^stned to be 
a Biaelor prirnacnrty .nd gi% - a hi=«£^. weighting, attL ±irf!r^^ a program's fui r*1anent of that 
vaifijable is njj^sanfiL- One way 01 -providing a — n^-' for a project on tme priority or 
impKit variable to ^Jt^orize a project on -^r incax into one of "JlirGfet categories, 
strong (3 points), jye^^aige (2 points ) , or weak ^ H>iint:) . In order to categorize 
projects, the valu_ f rt> " the in^ex and the categor- H>c''ndaries need to fca oatermrned 
through data or cx±'*^ estimtes. Tills project sought to determine the feasibility 

of defining categrr^^cs It/ obtaining distribution data ^or the indexes «Ki -osing the 
distribution data to establish category boundaries.. - aal Education Ag^<3=aL2iHs (LEAs) 
were surveyed :ra siit^di- ^^^a for each variable or^'^o give an indicaid.00^ futurs 
availability erf dai2i4» data that could be av^Ta^,"' e given advamiecf: rottlce, ar 

that the LEA cconsin^reP mpossible to collect. Oi= oth er outcome of thne srtudy is 
a dialogue between granfiag agencies and vocational etdmzatlon progrm (HlfTHCtors on 
the prtdorities in t^e ifxnr ' ing decisions and the impcrtti ace of providing data for 
evaluation of iunde£d3rro^ ams. 
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Related IResearch 

Davis and Salasin (19'''^> have summarized many of th- is^Mes in trie uh3 of eva niatlon 
results, including statements: by evaluators that thein: fi idiflgs are net ^ubr.ed and tizDse 
by adminiistrators that evalmation findings are not available vlien decz^ims have tr 
be iaade« Although there is anuch discussio' . of the neeid tco relate evailisarrzDn and 
decision nsaking, there have laeen few ef f or :s tc spec-i.fy the naaamer in v-nrizh this might 
occur. 

Edwards, Guttentag, arrri Snapper (197>'^ lave proposed aad applied a method called 
multi-attribute utility measnirement to as^-^^-^'^^ the Cfciice c}f ChiLd Develconssnt in 
defining tine major dimensions of importance" in devalopin^, -priorities for Ending 
research irrojects. The muiLti. -attribute u - .xty mea^remeirit method is one of a set c f 
mesthods ciLassified as decis i on aids by SI -vie, Fiscrioff, and Lichtensteir. (1977), a^ 
ognosed ±o formal behavior decision th^ry models. One fraiaework for x^e present 
sxrudy is provided by the pmcedures used "^lo aid decision:5=- 

A semnd framework is rhe problem dafining "i2njpi:Ct' for the rlv^^rse set of 
programs fxinded under the basic grants g^i--. in the Joh^.zloi^:! Ednucat:>..on hzt (VEA) . 
Bernstein: and Freeman (1975) define Impai.: -valiiarL^oi ais mv^ement or rsosinge toward 
the desirEd objectives of a project, i.e., jrHS3:ec :fie«i opeixationally lEE^imed goals 
and crit^iria of success. In the current vorJc these ^'/.'finltf ions are estsablished 
exteriEaOy to tdie individual projects. rrojecrcs cdl— aave varying goals and any one 
project laay ncz meet all the priorities of the fun dint progtam. 3y operationally 
defining the set of variables that define "hich iTJij^ii.T, anrd developing s scale or 
d^tTxauzlcm of these variables, individual projeictF - evaluated as '*lrxgh*' or 

"^ODw" Lnpacn: projects. 

A ii^larred study has been conducted by Otj . =tUr~ -s, and Clark a*^-. Ory, 

He rris, Tkieitt, and Clark (1978). Therv develoiiHe 3nd fieW test^ed a vocactonal 
eHfUcation evaluation model for progracis at the C'cssnnunity college level, bas.^d on 
SJH^iectJ-vet- and objectively derived data. V^eigiin? for six criteria were ^rcirived 
f rroHi psr^ red comparison ratings by state and local e-acarorcs^^ lec_-3lators, college 
tr':u>:rtee. and businessmen. Vocational programs we:^ assigned a ating s trong , 
ffMpgniati:. . or weak (3, 2, or 1 points) an each c s±x critevlm measures, ar.d the 
.xssESigort^s for the scale were established through ceonfeirer^ces in^^lving Itocal 
pff^iam administrators and the evaluators. 

In -ontrast, the present project has collected info rm^^it fata oa the ixstribution 
Qt inajor variables to establish scale categories. 



Methods Results 

In tJite first year's work on this project, noE--DV<*- Lapp highly relevant 

(impact^) dimensions were identified for the funding a^ ^cy, u : in^g interviews with 
decision laakers and review of the literature in vocatioi ol edi;C3?t:ion. In the current 
year th^e statements were revised in light of new le^i^ 'i^r^^ priorities. Three 
quest iomnsires were used. Questionnaire 1 provided for ranking and rating predictive 
and outoTOBE impact statements. Questionnaires 2 and 3 ooUrcted data^for the indexes 
for eacii predictive and outcome statement, respectively. 

The response rates for the different groups and gues cfliuma ar es ranged from 65% 
to 87%.' From the responses on the individual questiosnnal^— rp^umed it is clear that 



45 ItEA Questionnaires were mailed and 39 returned (02), 53 were mailed tc large 
cities a»d 42 returned (67%). Questionnaire 1 rankings are based on 38 LEA directors, 
33 respGwfents from large cities, and 5 state level STar6rvisors . 
O 
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most of the Infaxmatxon ssked for is either readily available or could be supplied if 
given aiivamce natrice. Ihrr most data, the SED can facilitate and improve the data base 
by condifflCtSaig wcarkshops or providing guidance to LEAs through sample forms and 
procedures^ The data reported in the questionnaires were highly variable in quality. 

Th« valiidity of the ±mpact statements as defined by the amount of agreement 
between' ratszis ras sig nifi cantly high as measured by Kendall's Coefficient of 
Concordance (W) . W was computed for the lanks given by three sets of judges: 
BOCES, Large City and SEE staff. W = .90 for the Predictive Impact Statements 
(X = 24.2. .01< p<.001), and W = .81 for the Outcome Statements (X^ = 21.8, ,01< 
p< .Odl). The SED staff rankings were used to develop weights, using the ranks 
similisr to the paired comparison method (Guilford, 1954). 

Alt index was developed for each impact variable. For example, one of the predictive 
impact statements that was given a high weighting was concerned with the number of 
employiHent options that would be available to students after being exposed to the 
program. The index for this impact statement was based on a listing for each 
pnogram of the occupational areas for which graduated of the program are to be pre- 
pared. The three scale categories of weak, average, and strong were defined by the 
distribution data as follows: if a program only offered 1 employment option, that 
program was given a weak rating; 2 3, 4, or 5 options, an average rating; more than 
5 employment options, a strong rating. 

Ancther high priority statement was that students will be trained for occupations 
where jobs are available. The index consisted of the ratio 

Total number ox local jobs available 

Total number of students expected to participate 

Intuitively, the fundfing applicant who shows that lc3s than one job per expected 
student is available Is a weaker applicant or is less likely to have an impact than an 
applicant for whom at least one job and less than two jobs is available (which would 
be considered average; . Stronger applicants provided data that more than two jobs 
were available in the local area for each student in the occupational program. 

By assigning a program to a category (weak, avera;;'3, or strong) on each of the 
indexes and then weighting the index by the importance of the impact statement it 
measures, a "score'' for each program, at both the funding and evaluation stage, is 
established. In addition" to the impact variables, grant proposals would be 
evaluated by screening criteria and quality rating scales in the areas of management/ 
planning and instruction/equipment. Since the variables that define high impact at 
both the funding and evaluation stages can be related, they are part of a system, 
providing feedback from evaluation to decision making. The basic process, then, 
has three major phases: 

i) Determination of whether a program has potential impact to decide if 

it v/ill be a) immediately funded, b) funded after recommended improvements 
are made, or c) rejected 
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2) Collection of evaluation data after a project is completed to assess outcome 
(Impact); and 

3) If a project is proposed for continuation, review of both outcome and predictive 
variables to make funding decisions. 

In addition to the general project review process, there should be provision for 
axmual review of the wei^ts and scale categories for both the predacrxve and outcome 
Impact statements. The process is Bayesian-like ±n procedures, but nmt formalized. 
The process includes using weights during one fiscal year, collectiiisg data on the 
impact statements after the project evaluation has occurred, and using the data for 
all projects to form distributions on the indexes for the impact stairements. These 
distributions can be examined to determine if the scale categories sudLll are 
appropriate or need to be "raised" or ""lowered" on the basis of project attainments. 
These processes should help in improving progracis and making funding decisions on t±e 
basis of both "objective" and "subjective" information. 



Educational Importance 

The signifixCTice of the present study is in demonstrating that evaluation 
findings and funding decisions can be linked to make better estimates of both predictive 
and outcome impact of projects. By showing that two groups of raters both concerned 
with the decision making process are in agreement on the important impact variables, 
the communication gap which presently exists bet:ween funding agencies, local education 
agencies and evaluators can be narrowed considerably. The process by which programs 
are both funded and evaluated is a potentially dynamic system, in the sense that 
the weights and indexes suggested in the study can constantly be reviewed and 
updated on the basis of current data. The nature of the decision making task is 
clarified and grant applicants would be aware that there were minimum standards as 
suggested by the scale categories. The complete procedure for linking funding 
decision making and evaluation is based on a series of objectively and subjectively 
knoxm data that are considered for revision on a systematic basis, and that provide 
standards known by both the decision maker and the applicant for funds. 
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^>He 1. Overall Ranking of P-redictive Umpact Statements 
by BOCES, Large City, arfd DOES Stsff 



Predictive^ Impact Statements Ranking 

La rsiie 

=?QCES ^ C i t^esT^ DOES a Qwerall 



1. Students wiTl be trained for 
occwpatia?^ where jobs are 
aval lable 




_ 


1 


3 


Project oaajectives are statsf^ 5 
In measurable terms 




3 


5 


3. No sex discrimination will 7 
made in recruiting and plac j 
students Fn vocational procr: 




7 


7 


k. Training objectives will be et 
in the most cost effective 
manner 


6 


6 


f 

k 


6 


5. Large nunaber of students v 1 10 
be trained 


9 10 


9 


6. Training will be provided ' 1 
increase ^ludents* employ^ 
ment ODtions 


3 


5.5 


2 


7. Students will be prepared ~ ; 
meet entry level skill re^ re- 
ments as specified by pro -a^r.t- 
ive employers, (e.g. emp lever's 
ratings of program perforrance 
objectives in terms of job rs- 
qulrernenits) . 


2 


1 


2 


1 


8. Program will serve students' 
interests. 


3 


k 


5.5 


k 


9. Program is articulated with 
local post secondary 
institutions 


8 


8 


8 


8 


10. Program will be replicable in 
other LEAs 


9 


: 10 


9 


10 



^Overall rank based on mean ranks. 
W= .896 

X^W (9 df) » 2k. ]S p < .01 
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Tabl« 2. Overall Rankings of Outcome Imract Statements 
by BOCES, Large City and DOES Staffs 



Outcome Impact Statements Ranking 



Large 

BOCES^ C 1 1 i es^ DOES^ Overal 1 



1. Program graduates are 
working in occupations 
for which they were 
trained . 


5 


2 


1 


1 


2. Project objectives 
are f ul f i 1 led 


6 


1 


5 


5 


3. No sex discrimination 
occurred in student 
selection, training, 
and Job placement 


7 


8 


7 


7 

1 


k. Training objectives are 
met in the most cost 
effective manner 


k 


6 


2 


6 


5. Large numbers of students 
are trained 


9 


9 


9 


9 


6. Training increases student 
employment options 


1 


3 


k 


2 


7. Employers are satisfied 
with graduates of program 


3 


5 


3 


3 


8. Students trained have 

positive attitudes toward 
work 


2 


k 


6 


1 


9. Students trained continue 
their education 


8 


7 


8 


8 


10. Program can be replicated 
in other LEAs 


10 


10 


10 


10 



Overall rank based on mean ranks. 



W - .809 

X^V/ (9 df) = 21 .84 p< .01 
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Project Quality Criterrii^ Predictive Impact 



Impact Statement 



1 . Students wi 1 1 be 
trained for occupations 
where jobs are available, 



2. Students will be pre- 
pared to meet entry level 
skill requirements of 
employers • 



3. Project objectives are 
stated in measurable terms. 



Scales 



sgor les 



4. Training objectives 
will be met in the most 
cost effective manner. 



WEAK 



A3/EKAGE 



STRONG 



WEIGHT 



SCALE 
X 

WEIGHT 



TotgJI nunnber of local jobs 

Tota*l nunnber of program students 

<1 U2 >2 

□ □ iZ] 

12 3X3 = 

No. gntry level skills checked by Employer 
No, skills listed by school 

□ Q 



□ 

1 2 3 

No. measurable objectives 
Total no. of objectives 



X 2 



0 

□ □ 
] 2 

Total grant $ 



1 

□ 
3 



X 2 



Total no. students served 

> $ 1 000 >$200 4 $ 1 000 ^ $200 

□ □ □ 
1 2 3 



X 1 



5. No Sex discrimination 
will be made in recruiting 
and placing students in 
vocational programs. 



6. Training will be pro- 
vided to increase students' 
employment options. 

7. Program will serve 
students* interests. 



No. sex discrimination items checked 
Total no. of items to check 



J.. 5 ^.5 ^.75 ^75 

□ □ □ 
1 2 3 

Number of employment options 

1 ^1 ^ 5 >5 

□ □ a 

1 2 3 



X 1 
(OE code) 

X 1 



Number of students to be served 

No. students selecting area 1st or 2nci 



<ll >1 42 

CZl □ 
1 2 



>2 
CZ] 



Total Impact Score: (Maximum 33) 
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